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DETAILED ACTION 
Claim Objections 

1 . Claim 10 is objected to because of the following informalities: line 1 states, "The method 
of claim 9 wherein the 1 wherein a video". It is unclear what applicant intended in the italicized 
portion. Appropriate correction is required. 

Claim Rejections - 35 USC §102 

2. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on 
sale in this country, more than one year prior to the date of application for patent in the United States. 

3. Claims 1-3, 5, 9, 1 1, 12, 16, 18 and 19 are rejected under 35 U.S.C. 102(b) as being 
anticipated by Shinohara et al. (USPN 5,619,619, newly cited, hereafter Shinohara). 

4. Regarding claim 1 , Shinohara discloses a method for determining affective information 
for at least one image in an imaging system, comprising the steps of: a) displaying a digital 
image (via display 223) for viewing by a user (Figure 22; column 23, line 55-65); b) monitoring 
the facial expression of the user (via TV camera 222) as the user views the digital image (column 
23, line 66-column 24, line 1, line 20-30); and c) using the facial expression of the user to 
determine affective (feeling) information for the digital image (column 24, line 1-12, line 31-42). 

5. Regarding claim 9, Shinohara discloses a method for providing affective information for 
images in an imaging system, comprising the steps of: a) sequentially displaying a plurality of 
digital images (inherently present in any personal computer with a display) for viewing by a user 
(Figure 22; column 23, line 55-65); b) monitoring the facial expression of the user (via TV 
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camera 222) as the user views each of the plurality of digital images (column 23, line 66-column 
24, line 1 , line 20-30); and c) using the facial expression of the user to determine affective 
(feeling) information (column 24, line 1-12, line 31-42). 

6. Regarding claim 1 1 , Shinohara discloses a system for providing affective information for 
images in an imaging system, comprising: a) a digital memory (RAM) which stores a set of 
digital images (inherent); b) a display 223 which sequentially displays the set of digital images 
(inherent) for viewing by a user (Figure 22; column 23, line 55-65); c) a camera 222 for 
automatically measuring the user's facial expression (column 23, line 66-column 24, line 1, line 
20-30); and d) a processor (computer 221, image processing board 237, neural network board 
238) for processing the signal from the camera to provide affective (feeling) information for the 
set of digital images (column 24, line 1-12, line 31-42). 

7. Regarding claim 2, Shinohara discloses the step of: d) associating the affective (feeling) 
information (extracting an operating instruction) with the digital image (column 24, line 1-12, 
line 31-51). 

8. Regarding claims 3 and 12, Shinohara discloses that a video camera is used to monitor 
the facial expression of the user (column 29, line 34-39). 

9. Regarding claim 5, Shinohara discloses that a plurality of digital images are displayed 
(inherent) for viewing by the user (Figure 22; column 23, line 55-65). 

10. Regarding claim 16, Shinohara discloses that the system further includes a sensor 
(camera) for measuring the user's physiology (face, eyes, eyebrows, mouth; column 13, line 16- 
46). 
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1 1 . Regarding claim 1 8, Shinohara discloses that the affective (feeling) information is stored 
in the digital memory (column 23, line 55-60; column 24, line 20-30, line 43-47). 

12. Regarding claim 19, Shinohara discloses that the affective (feeling) information is stored 
with each digital image in a digital image file (column 23, line 55-60; column 24, line 20-30, line 
43-47). 

13. The following is a quotation of the appropriate paragraphs of 35 U.S. C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(e) the invention was described in (1) an application for patent, published under section 122(b), by another filed 
in the United States before the invention by the applicant for patent or (2) a patent granted on an application for 
patent by another filed in the United States before the invention by the applicant for patent, except that an 
international application filed under the treaty defined in section 35 1 (a) shall have the effects for purposes of this 
subsection of an application filed in the United States only if the international application designated the United 
States and was published under Article 21(2) of such treaty in the English language. 

14. Claims 1-3,5,9, 11, 12, 16 and 18-20 are rejected under 35 U.S.C. 102(e) as being 
anticipated by Colmenarez et al. (USPAP 2003/0108241, newly cited, hereafter Colmenarez). 

15. Regarding claim 1, Colmenarez discloses a method for determining affective information 
for at least one image in an imaging system, comprising the steps of: a) displaying a digital 
image for viewing by a user (page 1 , paragraph 15); b) monitoring the facial expression of the 
user as the user views the digital image (page 1, paragraphs 16-17); and c) using the facial 
expression of the user to determine affective (mood/emotion) information for the digital image 
(page 1, paragraph 13; page 2, paragraphs 22 and 26). 

16. Regarding claim 9, Colmenarez discloses a method for providing affective information 
for images in an imaging system, comprising the steps of: a) sequentially (serially) displaying a 
plurality of digital images for viewing by a user (page 1, paragraph 15); b) monitoring the facial 
expression of the user as the user views each of the plurality of digital images (page 1 , 
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paragraphs 16-17); and c) using the facial expression of the user to determine affective 
(mood/emotion) information (page 1, paragraph 13; page 2, paragraphs 22 and 26). 

17. Regarding claim 11, Colmenarez discloses a system for providing affective information 
for images in an imaging system, comprising: a) a digital memory (storage area) which stores a 
set of digital images (page 1, paragraph 14); b) a display which sequentially (serially) displays 
the set of digital images for viewing by a user (page 1, paragraph 15); c) a camera 206 for 
automatically measuring the user's facial expression (page 1, paragraphs 16-17 and 20); and d) a 
processor 210 for processing the signal from the camera to provide affective (mood/emotion) 
information for the set of digital images (page 1, paragraphs 13 and 20; page 2, paragraphs 22 
and 26). 

18. Regarding claim 2, Colmenarez discloses the step of: d) associating the affective 
(mood/emotion) information with the digital image (page 1, paragraphs 13-14; page 2, paragraph 
30). 

19. Regarding claims 3 and 12, Colmenarez discloses that a video camera is used to monitor 
the facial expression of the user (page 1, paragraphs 1 1 and 20). 

20. Regarding claim 5, Colmenarez discloses that a plurality (set) of digital images are 
displayed for viewing by the user (page 1, paragraph 15). 

21. Regarding claim 16, Colmenarez discloses that the system further includes a sensor 
(camera) for measuring the user's physiology (position of the mouth and surrounding areas of the 
face; page 2, paragraph 25). 

22. Regarding claim 18, Colmenarez discloses that the affective (mood/emotion) information 
is stored in the digital memory (page 1, paragraphs 13-14; page 2, paragraphs 23 and 26). 
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23. Regarding claim 19, Colmenarez discloses that the affective (mood/emotion) information 
is stored with each digital image in a digital image file (page 1, paragraphs 13-14; page 2, 
paragraphs 23 and 26). 

24. Regarding claim 20, Colmenarez discloses that the digital image file includes affective 
(mood/emotion) information and user identifiers for a plurality of users (page 1, paragraphs 12- 
13; page 3, paragraph 34). 

Claim Rejections - 35 USC § 103 

25. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in - 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

26. Claim 17 is rejected under 35 U.S.C. 103(a) as being unpatentable over Shinohara or 
Colmenarez as applied to claim 16 above. 

27. Regarding claim 17, neither Shinohara nor Colmenarez disclose that the sensor for 
measuring the user's physiology measures the user's galvanic skin response. The examiner takes 
Official Notice that galvanic skin response is a well-known methodology for measuring a 
person's physiological response. It would have been obvious to one of ordinary skill in the art at 
the time the invention was made to measure the user's galvanic skin response in order to measure 
the user's physiology to help determine affective information. 

Allowable Subject Matter 
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28. Claims 4, 6-8, 10 and 13-15 are objected to as being dependent upon a rejected base 
claim, but would be allowable if rewritten in independent form including all of the limitations 
of the base claim and any intervening claims . 

29. The following is a statement of reasons for the indication of allowable subject matter: 
claims 4, 6-8, 10 and 13-15 recite limitations not found in the prior art of record including 
determining the smile size of the user while viewing a displayed image. 

Conclusion 

30. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. USPN 5,659,692 to Poggio et al. discloses a computer method and apparatus for 
video conferencing including parameters defining the degree of happiness/sadness of the mouth. 
USPN 6,046,773 to Martens et al. discloses an apparatus and method for decoding video images 
of an original video sequence consisting of a face with a mouth moving in various ways, 
representing smile, and changing intensity of its cheeks, representing blush. USPN 6,492,990 to 
Peleg et al. discloses a method for the automatic computerized audiovisual dubbing of movies 
wherein expression parameters include mouth width and smile. USPN 6,504,944 to Mihara et al. 
discloses an image recognition apparatus and method including recognizing a facial gesture or 
expression such as a smile or angry or distressed look. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Ryan J Hesseltine whose telephone number is 703-306-4069. 
The examiner can normally be reached on Monday - Friday, 8:30 AM - 5 PM. 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Amelia Au can be reached on 703-308-6604. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 

Ryan J. Hesseltine 
December 15, 2004 




